# Efficient Pre-training

Open-Qwen2VL is a multimodal model capable of receiving both images and text as input and generating text output.

A German-optimized large language model based on Meta Llama3-8B, continuously pre-trained on 65 billion German tokens, specifically optimized for German and supporting 32k long context

TinyLlama is a small language model with 1.1 billion parameters, adopting the same architecture and tokenizer as Llama 2, suitable for resource-constrained application scenarios.

VideoMAE is a video self-supervised pretraining model based on Masked Autoencoder (MAE), which learns internal video representations by predicting pixel values of masked video patches.

Chinese ELECTRA is a pre-trained model developed by the HIT-iFLYTEK Joint Lab based on Google's ELECTRA model, featuring a small parameter size but superior performance.

DistilCamemBERT is a distilled version of the French CamemBERT model, significantly reducing model complexity while maintaining performance through knowledge distillation techniques.

This model was pre-trained on a 250-million-word Chinese corpus using the MobileBERT architecture, with a training period of 15 days, completing 1 million iterations on a single A100 GPU.